[SQL][minor] use catalyst type converter in ScalaUdf#6182
[SQL][minor] use catalyst type converter in ScalaUdf#6182cloud-fan wants to merge 1 commit intoapache:masterfrom
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Use the same benchmark: import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.types._
case class Floor(child: Expression) extends UnaryExpression with Predicate {
override def foldable = child.foldable
def nullable = child.nullable
override def toString = s"Floor $child"
override def eval(input: Row): Any = {
child.eval(input) match {
case null => null
case ts: Int => ts - ts % 300
}
}
}
object T {
def benchmark(count: Int, expr: Expression): Unit = {
var i = 0
val row = new GenericRow(Array[Any](123, 21, 42))
val s = System.currentTimeMillis()
while (i < count) {
expr.eval(row)
i += 1
}
val e = System.currentTimeMillis()
println (s"${expr.getClass.getSimpleName} -- ${e - s} ms")
}
def main(args: Array[String]) {
def func(ts: Int) = ts - ts % 300
val udf0 = ScalaUdf(func _, IntegerType, BoundReference(0, IntegerType, true) :: Nil)
val udf1 = Floor(BoundReference(0, IntegerType, true))
benchmark(1000000, udf0)
benchmark(1000000, udf0)
benchmark(1000000, udf0)
benchmark(1000000, udf1)
benchmark(1000000, udf1)
benchmark(1000000, udf1)
}
}before: after: |
|
Test build #32813 has started for PR 6182 at commit |
|
Test build #32813 has finished for PR 6182 at commit
|
|
Merged build finished. Test PASSed. |
|
Test PASSed. |
|
LGTM; seems like a pretty straightforward optimization. I'm a bit new to Spark SQL, though, so I'll wait for committer with SQL experience to do the final sign-off on this. |
|
LGTM |
|
I am merging it to master and branch 1.4. |
It's a follow-up of #5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <cloud0fan@outlook.com> Closes #6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf (cherry picked from commit 2f22424) Signed-off-by: Yin Huai <yhuai@databricks.com>
It's a follow-up of apache#5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <cloud0fan@outlook.com> Closes apache#6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf
It's a follow-up of apache#5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <cloud0fan@outlook.com> Closes apache#6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf
It's a follow-up of apache#5154, we can speed up scala udf evaluation by create type converter in advance. Author: Wenchen Fan <cloud0fan@outlook.com> Closes apache#6182 from cloud-fan/tmp and squashes the following commits: 241cfe9 [Wenchen Fan] use converter in ScalaUdf
It's a follow-up of #5154, we can speed up scala udf evaluation by create type converter in advance.